test_runner: support custom message for expectFailure by Han5991 · Pull Request #61563 · nodejs/node

Han5991 · 2026-01-28T14:50:42Z

Summary

This PR enhances the expectFailure option in the test runner to accept different types of values, enabling both custom failure labels and robust error validation. This implementation is referenced from and inspired by nodejs/test-runner#10.

Changes

The expectFailure option now supports the following types:

String: Treated as a failure label (reason).

test('bug', { expectFailure: 'Investigating' }, ...);

RegExp / Function / Error Class: Treated as a matcher to validate the thrown error (similar to assert.throws).

test('regex', { expectFailure: /error message/ }, ...);
test('class', { expectFailure: RangeError }, ...);

Object:
- If it contains label or match properties, it's treated as a configuration object.
```
test('config', {
  expectFailure: {
    label: 'Known issue',
    match: /specific error/
  }
}, ...);
```
- Otherwise, it's treated as an object matcher (properties matching).

Inheritance:

Tests now inherit expectFailure from their parent suite. This allows marking an entire suite as expected to fail.

test('suite', { expectFailure: true }, async (t) => {
  await t.test('subtest', () => { throw new Error(); }); // Passes (expected failure)
});

References

Based on concepts from: doc(proposal): expectFailure label and/or matcher test-runner#10

Resolves: #61570

nodejs-github-bot · 2026-01-28T14:51:15Z

Review requested:

@nodejs/test_runner

doc/api/test.md

JakobJingleheimer

We already have a plan for the value of expectFail: it will soon accept a regular expression to match against.

Han5991 · 2026-01-28T15:25:42Z

We already gave a plan for the value of expectFail: it will soon accept a regular expression to match against.

@JakobJingleheimer
Ah, thanks for the context! I missed that there was an existing plan for error matching.

In that case, I'd be happy to pivot this PR to implement the expectFailure validation logic (accepting a string/regex to match the error) instead of just a message. Does that sound good, or is there someone else already working on it?"

JakobJingleheimer · 2026-01-28T15:33:11Z

@vassudanagunta you were part of the original discussion; did you happen to start an implementation?

To my knowledge though, no-one has started.

I had planned to pick it up next week, but if you would like to do, go ahead.

If you do, I think it would probably be better to start a new PR than to pivot this one. So open a draft and I'll add it to the test-runner team's kanban board so it gets proper visibility.

codecov · 2026-01-28T16:01:01Z

Codecov Report

❌ Patch coverage is 87.95181% with 10 lines in your changes missing coverage. Please review.
✅ Project coverage is 89.75%. Comparing base (da5efc4) to head (4c7bb37).
⚠️ Report is 26 commits behind head on main.

Files with missing lines	Patch %	Lines
lib/internal/test_runner/test.js	87.80%	10 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #61563      +/-   ##
==========================================
+ Coverage   88.84%   89.75%   +0.91%     
==========================================
  Files         674      674              
  Lines      204957   205682     +725     
  Branches    39309    39438     +129     
==========================================
+ Hits       182087   184607    +2520     
+ Misses      15088    13316    -1772     
+ Partials     7782     7759      -23

Files with missing lines	Coverage Δ
lib/internal/test_runner/reporter/tap.js	`94.88% <100.00%> (ø)`
lib/internal/test_runner/test.js	`96.85% <87.80%> (-0.49%)`	⬇️

... and 182 files with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

vassudanagunta · 2026-01-28T18:23:39Z

@JakobJingleheimer nope, haven't started this, though I had long ago implemented it in node-test-extra (not yet released).

I think it's important to get the requirements nailed. IMHO, #61570.

JakobJingleheimer · 2026-01-28T20:10:19Z

As I said, let's put together a proposal in the nodejs/test_runner repo 🙂

ljharb · 2026-01-28T22:06:07Z

reviewed before reading the discussion; imo a string should work as in this PR whether or not it also supports accepting a regex.

JakobJingleheimer · 2026-01-28T22:31:51Z

It could do. My concern is supporting this without considering the intended regex feature accidentally precluding that intended feature, or inadvertently creating a breaking change, or creating heavily conflicting PRs (very frustrating for the implementators).

I think we can likely get both; we can easily avoid those problems with a quick proposal so everyone is on the same page 🙂

I should start a discussion in that repo?

Please start a proposal like the ones already in that repo 🙂 https://github.com/nodejs/test-runner/tree/main/proposals we can discuss it in that PR

ljharb · 2026-01-28T22:33:40Z

conflicts fair; as long as the "should expect failure" uses truthiness (does an empty string count as true or false, though?), i can't foresee any semantic collision.

Han5991 · 2026-01-29T02:26:17Z

@ljharb @vassudanagunta

I've opened a proposal PR in the test-runner repository as suggested by @JakobJingleheimer.
It would be great to continue the discussion on the spec details there:
nodejs/test-runner#10

doc/api/test.md

vassudanagunta

I think this needs to be documented a little better for the user.

doc/api/test.md

JakobJingleheimer · 2026-02-04T18:26:45Z

@Han5991 the proposal isn't finished / accepted yet (still hasn't been reviewed by the rest of the test-runner team), so I think it's premature to resume this (the proposal isn't a requirement, but I think it's a good idea and will reduce churn, needless re-reviews, etc—and indeed, there was just earlier today another adjustment to align terms). I do appreciate the enthusiasm 😁

It's added to the team's agenda, so it'll get raised at the next meeting.

nodejs-github-bot · 2026-02-25T22:03:56Z

CI: https://ci.nodejs.org/job/node-test-pull-request/71461/

vassudanagunta · 2026-02-25T22:55:06Z

think what you're now proposing is undue maintenance burden

Actually all I am proposing is that the decision be made "eyes wide open", with implications in full view. While API design is something I care a lot about, I understand this API so it's no skin off my back whichever way it goes.

nodejs-github-bot · 2026-02-26T09:15:39Z

CI: https://ci.nodejs.org/job/node-test-pull-request/71464/

nodejs-github-bot · 2026-02-26T10:15:52Z

Commit Queue failed

- Loading data for nodejs/node/pull/61563
✔  Done loading data for nodejs/node/pull/61563
----------------------------------- PR info ------------------------------------
Title      test_runner: support custom message for expectFailure (#61563)
Author     sangwook <rewq5991@gmail.com> (@Han5991)
Branch     Han5991:test-runner/support-getxfail-message -> nodejs:main
Labels     needs-ci, commit-queue-squash, test_runner
Commits    2
 - test_runner: expose expectFailure message
 - test_runner: fix empty object test assert logic
Committers 1
 - sangwook <rewq5991@gmail.com>
PR-URL: https://github.com/nodejs/node/pull/61563
Fixes: https://github.com/nodejs/node/issues/61570
Reviewed-By: Jordan Harband <ljharb@gmail.com>
Reviewed-By: Aviv Keller <me@aviv.sh>
Reviewed-By: Jacob Smith <jacob@frende.me>
------------------------------ Generated metadata ------------------------------
PR-URL: https://github.com/nodejs/node/pull/61563
Fixes: https://github.com/nodejs/node/issues/61570
Reviewed-By: Jordan Harband <ljharb@gmail.com>
Reviewed-By: Aviv Keller <me@aviv.sh>
Reviewed-By: Jacob Smith <jacob@frende.me>
--------------------------------------------------------------------------------
   ℹ  This PR was created on Wed, 28 Jan 2026 14:50:42 GMT
   ✔  Approvals: 3
   ✔  - Jordan Harband (@ljharb): https://github.com/nodejs/node/pull/61563#pullrequestreview-3719402319
   ✔  - Aviv Keller (@avivkeller): https://github.com/nodejs/node/pull/61563#pullrequestreview-3723610081
   ✔  - Jacob Smith (@JakobJingleheimer): https://github.com/nodejs/node/pull/61563#pullrequestreview-3857218091
   ✘  Last GitHub CI failed
   ℹ  Last Full PR CI on 2026-02-26T09:15:39Z: https://ci.nodejs.org/job/node-test-pull-request/71464/
- Querying data for job/node-test-pull-request/71464/
✔  Build data downloaded
   ✔  Last Jenkins CI successful
--------------------------------------------------------------------------------
   ✔  Aborted `git node land` session in /home/runner/work/node/node/.ncu

https://github.com/nodejs/node/actions/runs/22437607383

Han5991 · 2026-02-26T11:51:35Z

@JakobJingleheimer

It seems like github ci failed. Should I try running it again with rebase?

JakobJingleheimer · 2026-02-26T12:06:54Z

CI didn't actually fail though. Earlier this morning I asked one of the CI maintainers; I'm waiting to hear back.

nodejs-github-bot · 2026-02-26T15:37:58Z

Landed in 2e1265a

This change exposes the expectFailure message in the test runner and adds edge cases for expectFailure ambiguity. PR-URL: #61563 Fixes: #61570 Reviewed-By: Jordan Harband <ljharb@gmail.com> Reviewed-By: Aviv Keller <me@aviv.sh> Reviewed-By: Jacob Smith <jacob@frende.me>

nodejs-github-bot added needs-ci PRs that need a full CI run. test_runner Issues and PRs related to the test runner subsystem. labels Jan 28, 2026

aduh95 reviewed Jan 28, 2026

View reviewed changes

doc/api/test.md Outdated Show resolved Hide resolved

JakobJingleheimer previously requested changes Jan 28, 2026

View reviewed changes

This comment was marked as resolved.

Sign in to view

ljharb approved these changes Jan 28, 2026

View reviewed changes

Han5991 force-pushed the test-runner/support-getxfail-message branch from 1af3584 to 13aedaa Compare January 28, 2026 23:26

Han5991 requested a review from aduh95 January 28, 2026 23:28

Han-sangwook mentioned this pull request Jan 29, 2026

feat: proposal for expectFailure enhancements nodejs/test-runner#9

Closed

Han5991 mentioned this pull request Jan 29, 2026

doc(proposal): expectFailure label and/or matcher nodejs/test-runner#10

Merged

avivkeller approved these changes Jan 29, 2026

View reviewed changes

vassudanagunta reviewed Jan 29, 2026

View reviewed changes

doc/api/test.md Show resolved Hide resolved

Han5991 marked this pull request as draft January 31, 2026 07:56

Han5991 closed this Jan 31, 2026

Han5991 reopened this Feb 3, 2026

Han5991 marked this pull request as ready for review February 3, 2026 22:45

Han5991 requested a review from JakobJingleheimer February 3, 2026 22:45

Han5991 force-pushed the test-runner/support-getxfail-message branch from 061f049 to 346ec8f Compare February 4, 2026 00:49

vassudanagunta reviewed Feb 4, 2026

View reviewed changes

doc/api/test.md Outdated Show resolved Hide resolved

doc/api/test.md Outdated Show resolved Hide resolved

doc/api/test.md Show resolved Hide resolved

doc/api/test.md Outdated Show resolved Hide resolved

doc/api/test.md Outdated Show resolved Hide resolved

JakobJingleheimer added the request-ci Add this label to start a Jenkins CI on a PR. label Feb 25, 2026

github-actions bot added request-ci-failed An error occurred while starting CI via request-ci label, and manual interventon is needed. and removed request-ci Add this label to start a Jenkins CI on a PR. labels Feb 25, 2026

This comment was marked as resolved.

Sign in to view

JakobJingleheimer approved these changes Feb 25, 2026

View reviewed changes

JakobJingleheimer added request-ci Add this label to start a Jenkins CI on a PR. and removed request-ci-failed An error occurred while starting CI via request-ci label, and manual interventon is needed. labels Feb 25, 2026

github-actions bot removed the request-ci Add this label to start a Jenkins CI on a PR. label Feb 25, 2026

JakobJingleheimer added commit-queue Add this label to land a pull request using GitHub Actions. commit-queue-squash Add this label to instruct the Commit Queue to squash all the PR commits into the first one. labels Feb 26, 2026

nodejs-github-bot added commit-queue-failed An error occurred while landing this pull request using GitHub Actions. and removed commit-queue Add this label to land a pull request using GitHub Actions. labels Feb 26, 2026

This comment was marked as outdated.

Sign in to view

JakobJingleheimer added commit-queue Add this label to land a pull request using GitHub Actions. and removed commit-queue-failed An error occurred while landing this pull request using GitHub Actions. labels Feb 26, 2026

nodejs-github-bot added commit-queue-failed An error occurred while landing this pull request using GitHub Actions. and removed commit-queue Add this label to land a pull request using GitHub Actions. labels Feb 26, 2026

JakobJingleheimer added commit-queue Add this label to land a pull request using GitHub Actions. and removed commit-queue-failed An error occurred while landing this pull request using GitHub Actions. labels Feb 26, 2026

nodejs-github-bot removed the commit-queue Add this label to land a pull request using GitHub Actions. label Feb 26, 2026

nodejs-github-bot merged commit 2e1265a into nodejs:main Feb 26, 2026
96 of 100 checks passed

github-actions bot mentioned this pull request Mar 2, 2026

2026-03-03, Version 25.8.0 (Current) #62073

Open

Uh oh!

Conversation

Han5991 commented Jan 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Changes

References

Uh oh!

nodejs-github-bot commented Jan 28, 2026

Uh oh!

Uh oh!

JakobJingleheimer left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Han5991 commented Jan 28, 2026

Uh oh!

JakobJingleheimer commented Jan 28, 2026

Uh oh!

codecov bot commented Jan 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

vassudanagunta commented Jan 28, 2026

Uh oh!

JakobJingleheimer commented Jan 28, 2026

Uh oh!

This comment was marked as resolved.

ljharb commented Jan 28, 2026

Uh oh!

JakobJingleheimer commented Jan 28, 2026

Uh oh!

ljharb commented Jan 28, 2026

Uh oh!

Han5991 commented Jan 29, 2026

Uh oh!

Uh oh!

vassudanagunta left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

JakobJingleheimer commented Feb 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

This comment was marked as resolved.

nodejs-github-bot commented Feb 25, 2026

Uh oh!

vassudanagunta commented Feb 25, 2026

Uh oh!

This comment was marked as outdated.

nodejs-github-bot commented Feb 26, 2026

Uh oh!

nodejs-github-bot commented Feb 26, 2026

Uh oh!

Han5991 commented Feb 26, 2026

Uh oh!

JakobJingleheimer commented Feb 26, 2026

Uh oh!

Uh oh!

nodejs-github-bot commented Feb 26, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

Han5991 commented Jan 28, 2026 •

edited

Loading

JakobJingleheimer left a comment •

edited

Loading

codecov bot commented Jan 28, 2026 •

edited

Loading

JakobJingleheimer commented Feb 4, 2026 •

edited

Loading